[SPARK-61102][CONNECT][PYTHON] Match PySpark NumPy scalar-class checks - #381

What changes were proposed in this pull request?

Apply PySpark 4.3.0's numeric scalar-class checks after NumPy dtype inference. Distinct aliases such as np.longlong or an eight-byte np.longdouble now raise PySpark's bare AssertionError when their class is rejected. Supported classes retain their literal types, and wider unsupported dtypes retain their structured errors. Update scalar and array regressions for platform-dependent aliases.

Why are the changes needed?

SPARK-61102. Dtype-kind and width checks alone accept classes that official PySpark rejects, including longdouble on macOS arm64.

Does this PR introduce any user-facing change?

Yes. Scalar and array literals apply PySpark's class validation as well as dtype inference.

How was this patch tested?

• Formatting, Rust core/API units, integration-test compilation with and without WASM, and native-extension Python suites: 2,170 passed, eight live-server skips.
• Twenty no-server scalar/array expectation checks against official PySpark 4.3.0. Two native live cases passed with verified imports, covering three supported NumPy schema types and existing listener behavior; this is not exhaustive NumPy value coverage.
• API inspection found no missing inspected members across 12 class surfaces. This checks member presence; signatures and behavior differ in places, including the separate Row tuple gap.
• Full raw transport on the default four-thread fixture: 87 normal files, 2,586 passed, 641 skipped, one deselected, no failures/errors/retries. Eleven files were entirely skipped. The existing metadata-column manifest entry failed its separate advisory check because the test catalog plugin was unavailable (one failure, 57 deselections).
• Separate local-cluster resource/artifact phase: 15 passing invocations across four modules (ten substantive cases and five helpers), 14 JVM-guarded skips, no failures/errors/retries.

The original full run's seeded-random failure remains recorded: the unchanged test failed identically in official Python and both saved Rust transports on two threads, then passed in all three on its default four-thread fixture. Earlier CPU-queue expirations executed no checks.

The full/cluster suites use official Python converters over this build's Rust raw transport; they do not establish native high-level API, native ReleaseSession or native artifact-upload parity. They lack per-child import attestation. Server logs retain deliberate negative-test diagnostics and shutdown races; direct successful shutdown was recorded without final descendant-process telemetry.

Clippy remains advisory exit 101. Linux and Windows were not run locally.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex

This pull request and its description were written by OpenAI Codex.